Detecting Duplicate Job Ads

Importing Libraries
Loading Data
Tokenizing
Showing the Differences Between Ads
Detecting Identical Ads
Detecting Approximate Duplicate Ads
Detecting Duplicates at Scale
Using duplicates

This is an exploration of various techniques for finding duplicate texts using job ads from the Adzuna Salary Prediction Kaggle Competition.

See the overview article for details.

Importing Libraries

import pandas as pd
import re

from IPython.display import HTML, display

We’ll define a type TokenList for convenience

from typing import List, Any, Callable, Tuple, Union

Token = str
TokenList = List[Token]

pd.options.display.max_colwidth=100

Load in the Data

Get the data from Adzunda Job Salary Prediction Kaggle Competition, put it in the data subfolder and unzip all the files.

You can do this manually, or use the Kaggle API (once you’ve installed the API, downloaded your kaggle.json file and agreed to the competition rules)

# for split, ext in [('Test', 'zip'), ('Train', 'zip'), ('Valid', 'csv')]:
#     !kaggle competitions download -c job-salary-prediction --path data/ -f {split}_rev1.{ext}
    
# !find data/ -name '*.zip' -execdir unzip '{}' ';'
# !find data/ -name '*.zip' -exec rm '{}' ';'

# !ls data/

%%time
dfs = []
for split in ['Train', 'Valid', 'Test']:
    dfs.append(pd.read_csv(f'data/{split}_rev1.csv').assign(split=split))
df = pd.concat(dfs, sort=False, ignore_index=True)
del dfs

CPU times: user 10.7 s, sys: 4.19 s, total: 14.9 s
Wall time: 17.6 s

ads = list(df.FullDescription)

Tokenizing

Some functions to split text into “tokens” or “sentences”.

Note that this is a lossy function because we lose extra whitespace, but in this application it’s not very intestesting

These are very simple at the moment; we could refactor them to improve things slightly.

whitespace = re.compile('\s+')
end_sentence = re.compile('[.!?]\s+')

def tokenize(s:str) -> TokenList:
    '''Split a string into tokens'''
    return whitespace.split(s)

def untokenize(ts:TokenList) -> str:
    '''Join a list of tokens into a string'''
    return ' '.join(ts)

def sentencize(s:str) -> TokenList:
    '''Split a string into a list of sentences'''
    return end_sentence.split(s)

def unsentencise(ts:TokenList) -> str:
    '''Join a list of sentences into a string'''
    return '. '.join(ts)

def html_unsentencise(ts:TokenList) -> str:
    '''Joing a list of sentences into HTML for display'''
    return ''.join(f'<p>{t}</p>' for t in ts)

Showing Differences between ads

It’s very hard to spot the difference between two ad texts, which will make it hard to evaluate how our duplicate detection is doing.

We’ll develop a method for highlighting the differences and showing the ads side by side.

See the article for a summary.

ad_a, ad_b = ads[2647], ads[4408]
print(ad_a)
print()
print(ad_b)

Java J****EE Developer  ****k  ****k  Music, Film & TV  London Java J****EE Developers required for software house with client sectors of music, film and TV. Salary: Maximum ****: Discretionary bonus and benefits package. Location: Near Euston and King's Cross, London THE COMPANY: Consistent new business wins for the world leader in the provision of software solutions to the Music and Entertainment industry has given rise to the need for an experienced Java Developer. The working environment here is very pleasant with a casual dress code, laid back and friendly atmosphere, but also hardworking and dynamic with the autonomy to drive your job role forward. This is predominantly a development role, but you will be involved in the full product life cycle including design and clientfacing duties, so they need a good allrounder. EXPERIENCE REQUIRED: The experience required for this role is as follows:  A minimum of 5 years experience in the development of web applications for the J****EE development platform.  A minimum of 5 years experience in Java  Strong knowledge in all of JSP, Servlet, JDBC, JavaScript, SQL and HTML technologies.  Good knowledge of CSS, XML and DHTML  A personality suited to clientfacing situations  good communication skills.  A good standard of written English The above experience is essential. You require all of the above experience in order for to be eligible for this role. The following experience is desirable, though not essential:  Knowledge of the WebSphere development environment and Application Server.  Knowledge of and experience with AJAX (Asynchronous JavaScript XML)  Experience with IBM DB**** THE ROLE: This is a full SDLC role: You will be involved in all stages of the software development cycle from requirements gathering and specification through development, implementation, QA and support. You'll be involved in different technologies across the board from Front Office to Back Office. Please note there is no Spring or Hibernate: The company have instead developed their own inhouse frameworks. THE OPPORTUNITY Why work here? As for prospects, where you can take this role is flexible, as the role entails a wide remit across most aspects of development. Therefore, if you wish, you could become more clientfacing and progress to what is essentially a business analyst role, or you may wish to specialise more on the technical side of things and push the boundaries of the technology. This is a central role that essentially can take off in any direction. Here, you will have enough autonomy to define your own role. Therefore, if you take the initiative you can shape your role for the future and drive your own progression. Overall, this is a lovely place to work  it's a privatelyowned company and feels more like a family company, not at all institutionalised  everyone has a stake, everyone has a say. Being music, entertainment and film it's an interesting industry to work in too, with a wide range of clients both local and foreign. Location: Near Euston and King's Cross, London

NEW  Java J****EE Developer – ****k  ****k  Music, Film TV  London Java J****EE Developers required for software house with client sectors of music, film and TV. Salary: Maximum ****: Discretionary bonus and benefits package. Location: Near Euston and King’s Cross, London THE COMPANY: Consistent new business wins for the world leader in the provision of software solutions to the Music and Entertainment industry has given rise to the need for an experienced Java Developer. The working environment here is very pleasant with a casual dress code, laid back and friendly atmosphere, but also hardworking and dynamic with the autonomy to drive your job role forward. This is predominantly a development role, but you will be involved in the full product lifecycle including design and clientfacing duties, so they need a good allrounder. EXPERIENCE REQUIRED: The experience required for this role is as follows: A minimum of 5 years experience in the development of web applications for the J****EE development platform. A minimum of 5 years experience in Java Strong knowledge in all of JSP, Servlet, JDBC, JavaScript, SQL and HTML technologies. Good knowledge of CSS, XML and DHTML A personality suited to clientfacing situations  good communication skills. A good standard of written English The above experience is essential. You require all of the above experience in order for to be eligible for this role. The following experience is desirable, though not essential: Knowledge of the Websphere development environment and application server. Knowledge of and experience with AJAX (Asynchronous JavaScript XML) Experience with IBM DB**** THE ROLE: This is a full SDLC role: You will be involved in all stages of the software development cycle from requirements gathering and specification through development, implementation, QA and support. You'll be involved in different technologies across the board from front office to back office. Please note there is no Spring or Hibernate: The company have instead developed their own inhouse frameworks. THE OPPORTUNITY: Why work here? As for prospects, where you can take this role is flexible, as the role entails a wide remit across most aspects of development. Therefore, if you wish, you could become more clientfacing and progress to what is essentially a business analyst role, or you may wish to specialise more on the technical side of things and push the boundaries of the technology. This is a central role that essentially can take off in any direction. Here, you will have enough autonomy to define your own role. Therefore, if you take the initiative you can shape your role for the future and drive your own progression. Overall, this is a lovely place to work  it's a privatelyowned company and feels more like a family company, not at all institutionalised  everyone has a stake, everyone has a say. Being music, entertainment and film it's an interesting industry to work in too, with a wide range of clients both local and foreign. Location: Near Euston and King’s Cross, London This job was originally posted as www.cwjobs.co.uk/JobSeeking/JavaJ****EEDeveloper****k****kMusicFilmTVLondon_job****

Breaking it into sentences makes is a bit easier to read the ads, but it’s still hard to compare them.

HTML(html_unsentencise(sentencize(ad_a)))

Java J****EE Developer ****k ****k Music, Film & TV London Java J****EE Developers required for software house with client sectors of music, film and TV

Salary: Maximum ****: Discretionary bonus and benefits package

Location: Near Euston and King's Cross, London THE COMPANY: Consistent new business wins for the world leader in the provision of software solutions to the Music and Entertainment industry has given rise to the need for an experienced Java Developer

The working environment here is very pleasant with a casual dress code, laid back and friendly atmosphere, but also hardworking and dynamic with the autonomy to drive your job role forward

This is predominantly a development role, but you will be involved in the full product life cycle including design and clientfacing duties, so they need a good allrounder

EXPERIENCE REQUIRED: The experience required for this role is as follows: A minimum of 5 years experience in the development of web applications for the J****EE development platform

A minimum of 5 years experience in Java Strong knowledge in all of JSP, Servlet, JDBC, JavaScript, SQL and HTML technologies

Good knowledge of CSS, XML and DHTML A personality suited to clientfacing situations good communication skills

A good standard of written English The above experience is essential

You require all of the above experience in order for to be eligible for this role

The following experience is desirable, though not essential: Knowledge of the WebSphere development environment and Application Server

Knowledge of and experience with AJAX (Asynchronous JavaScript XML) Experience with IBM DB**** THE ROLE: This is a full SDLC role: You will be involved in all stages of the software development cycle from requirements gathering and specification through development, implementation, QA and support

You'll be involved in different technologies across the board from Front Office to Back Office

Please note there is no Spring or Hibernate: The company have instead developed their own inhouse frameworks

THE OPPORTUNITY Why work here

As for prospects, where you can take this role is flexible, as the role entails a wide remit across most aspects of development

Therefore, if you wish, you could become more clientfacing and progress to what is essentially a business analyst role, or you may wish to specialise more on the technical side of things and push the boundaries of the technology

This is a central role that essentially can take off in any direction

Here, you will have enough autonomy to define your own role

Therefore, if you take the initiative you can shape your role for the future and drive your own progression

Overall, this is a lovely place to work it's a privatelyowned company and feels more like a family company, not at all institutionalised everyone has a stake, everyone has a say

Being music, entertainment and film it's an interesting industry to work in too, with a wide range of clients both local and foreign

Location: Near Euston and King's Cross, London

HTML(html_unsentencise(sentencize(ad_b)))

NEW Java J****EE Developer – ****k ****k Music, Film TV London Java J****EE Developers required for software house with client sectors of music, film and TV

Salary: Maximum ****: Discretionary bonus and benefits package

Location: Near Euston and King’s Cross, London THE COMPANY: Consistent new business wins for the world leader in the provision of software solutions to the Music and Entertainment industry has given rise to the need for an experienced Java Developer

The working environment here is very pleasant with a casual dress code, laid back and friendly atmosphere, but also hardworking and dynamic with the autonomy to drive your job role forward

This is predominantly a development role, but you will be involved in the full product lifecycle including design and clientfacing duties, so they need a good allrounder

EXPERIENCE REQUIRED: The experience required for this role is as follows: A minimum of 5 years experience in the development of web applications for the J****EE development platform

A minimum of 5 years experience in Java Strong knowledge in all of JSP, Servlet, JDBC, JavaScript, SQL and HTML technologies

Good knowledge of CSS, XML and DHTML A personality suited to clientfacing situations good communication skills

A good standard of written English The above experience is essential

You require all of the above experience in order for to be eligible for this role

The following experience is desirable, though not essential: Knowledge of the Websphere development environment and application server

You'll be involved in different technologies across the board from front office to back office

Please note there is no Spring or Hibernate: The company have instead developed their own inhouse frameworks

THE OPPORTUNITY: Why work here

As for prospects, where you can take this role is flexible, as the role entails a wide remit across most aspects of development

This is a central role that essentially can take off in any direction

Here, you will have enough autonomy to define your own role

Therefore, if you take the initiative you can shape your role for the future and drive your own progression

Overall, this is a lovely place to work it's a privatelyowned company and feels more like a family company, not at all institutionalised everyone has a stake, everyone has a say

Being music, entertainment and film it's an interesting industry to work in too, with a wide range of clients both local and foreign

Location: Near Euston and King’s Cross, London This job was originally posted as www.cwjobs.co.uk/JobSeeking/JavaJ****EEDeveloper****k****kMusicFilmTVLondon_job****

Highlighting the Differences

We can use Python’s difflib to find and highlight differences between the two texts.

import difflib
import html

We can markup the differences using HTML; we’ll colour and bold each token.

We could mark each token:

def mark_text(text:str) -> str:
    return f'<span style="color: red;">{text}</span>'

def mark_span(text:TokenList) -> TokenList:
    return [mark_text(token) for token in text]

or the whole span:

def mark_span(text:TokenList) -> TokenList:
    if len(text) > 0:
        text[0] = '<span style="background: #69E2FB;">' + text[0]
        text[-1] += '</span>'
    return text

Identity function for convenience

def identity(x):
    return x

markup_diff processes the tokenlists, marking any tokens that are different.

To understand autojunk=False see Dear DiffLib; it’s turning off some clever magic I don’t want to use.

Note: The output should actually be something like a MakedUpTokenList, but I’m not really sure about the abstraction I would want to use.

def markup_diff(a:TokenList, b:TokenList,
                mark:Callable[[TokenList], TokenList]=mark_span,
                mark_equal: Callable[[TokenList], TokenList]=identity,
                isjunk:Union[None, Callable[[Token], bool]]=None) -> Tuple[TokenList, TokenList]:
    """Returns a and b with any differences processed by mark
    
    Junk is ignored by the differ
    """
    seqmatcher = difflib.SequenceMatcher(isjunk=isjunk, a=a, b=b, autojunk=False)
    out_a, out_b = [], []
    for tag, a0, a1, b0, b1 in seqmatcher.get_opcodes():
        markup = mark_equal if tag == 'equal' else mark
        out_a += markup(a[a0:a1])
        out_b += markup(b[b0:b1])
    assert len(out_a) == len(a)
    assert len(out_b) == len(b)
    return out_a, out_b

Make sure to escape HTML special characters because our markup is in HTML

tok_a, tok_b = tokenize(html.escape(ad_a)), tokenize(html.escape(ad_b))

out_a, out_b = markup_diff(tok_a, tok_b)

It’s now clear that there are some small differences in punctuation and whitespace between the two ads, but are otherwise the same (except for the last line).

HTML(untokenize(out_a))

Java J****EE Developer ****k ****k Music, Film & TV London Java J****EE Developers required for software house with client sectors of music, film and TV. Salary: Maximum ****: Discretionary bonus and benefits package. Location: Near Euston and King's Cross, London THE COMPANY: Consistent new business wins for the world leader in the provision of software solutions to the Music and Entertainment industry has given rise to the need for an experienced Java Developer. The working environment here is very pleasant with a casual dress code, laid back and friendly atmosphere, but also hardworking and dynamic with the autonomy to drive your job role forward. This is predominantly a development role, but you will be involved in the full product life cycle including design and clientfacing duties, so they need a good allrounder. EXPERIENCE REQUIRED: The experience required for this role is as follows: A minimum of 5 years experience in the development of web applications for the J****EE development platform. A minimum of 5 years experience in Java Strong knowledge in all of JSP, Servlet, JDBC, JavaScript, SQL and HTML technologies. Good knowledge of CSS, XML and DHTML A personality suited to clientfacing situations good communication skills. A good standard of written English The above experience is essential. You require all of the above experience in order for to be eligible for this role. The following experience is desirable, though not essential: Knowledge of the WebSphere development environment and Application Server. Knowledge of and experience with AJAX (Asynchronous JavaScript XML) Experience with IBM DB**** THE ROLE: This is a full SDLC role: You will be involved in all stages of the software development cycle from requirements gathering and specification through development, implementation, QA and support. You'll be involved in different technologies across the board from Front Office to Back Office. Please note there is no Spring or Hibernate: The company have instead developed their own inhouse frameworks. THE OPPORTUNITY Why work here? As for prospects, where you can take this role is flexible, as the role entails a wide remit across most aspects of development. Therefore, if you wish, you could become more clientfacing and progress to what is essentially a business analyst role, or you may wish to specialise more on the technical side of things and push the boundaries of the technology. This is a central role that essentially can take off in any direction. Here, you will have enough autonomy to define your own role. Therefore, if you take the initiative you can shape your role for the future and drive your own progression. Overall, this is a lovely place to work it's a privatelyowned company and feels more like a family company, not at all institutionalised everyone has a stake, everyone has a say. Being music, entertainment and film it's an interesting industry to work in too, with a wide range of clients both local and foreign. Location: Near Euston and King's Cross, London

HTML(untokenize(out_b))

NEW Java J****EE Developer – ****k ****k Music, Film TV London Java J****EE Developers required for software house with client sectors of music, film and TV. Salary: Maximum ****: Discretionary bonus and benefits package. Location: Near Euston and King’s Cross, London THE COMPANY: Consistent new business wins for the world leader in the provision of software solutions to the Music and Entertainment industry has given rise to the need for an experienced Java Developer. The working environment here is very pleasant with a casual dress code, laid back and friendly atmosphere, but also hardworking and dynamic with the autonomy to drive your job role forward. This is predominantly a development role, but you will be involved in the full product lifecycle including design and clientfacing duties, so they need a good allrounder. EXPERIENCE REQUIRED: The experience required for this role is as follows: A minimum of 5 years experience in the development of web applications for the J****EE development platform. A minimum of 5 years experience in Java Strong knowledge in all of JSP, Servlet, JDBC, JavaScript, SQL and HTML technologies. Good knowledge of CSS, XML and DHTML A personality suited to clientfacing situations good communication skills. A good standard of written English The above experience is essential. You require all of the above experience in order for to be eligible for this role. The following experience is desirable, though not essential: Knowledge of the Websphere development environment and application server. Knowledge of and experience with AJAX (Asynchronous JavaScript XML) Experience with IBM DB**** THE ROLE: This is a full SDLC role: You will be involved in all stages of the software development cycle from requirements gathering and specification through development, implementation, QA and support. You'll be involved in different technologies across the board from front office to back office. Please note there is no Spring or Hibernate: The company have instead developed their own inhouse frameworks. THE OPPORTUNITY: Why work here? As for prospects, where you can take this role is flexible, as the role entails a wide remit across most aspects of development. Therefore, if you wish, you could become more clientfacing and progress to what is essentially a business analyst role, or you may wish to specialise more on the technical side of things and push the boundaries of the technology. This is a central role that essentially can take off in any direction. Here, you will have enough autonomy to define your own role. Therefore, if you take the initiative you can shape your role for the future and drive your own progression. Overall, this is a lovely place to work it's a privatelyowned company and feels more like a family company, not at all institutionalised everyone has a stake, everyone has a say. Being music, entertainment and film it's an interesting industry to work in too, with a wide range of clients both local and foreign. Location: Near Euston and King’s Cross, London This job was originally posted as www.cwjobs.co.uk/JobSeeking/JavaJ****EEDeveloper****k****kMusicFilmTVLondon_job****

Putting Ads Side by Side

It’s still hard to compare two ads below each other, so let’s try to put them side by side.

We could in fact show differences the ads at a sentence level instead of a word level

sent_a, sent_b = sentencize(html.escape(ad_a)), sentencize(html.escape(ad_b))
out_a, out_b = markup_diff(sent_a, sent_b)

HTML(html_unsentencise(out_a))